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ABSTRACT 


Desertbes design and use of recording quizboard which 
records right and wrong answers to questions based on 
visttor center exhibits. Thts helps determine how well 
exhibit messages are reaching visttors. Initial results 
tndicate that taped messages are more effective than those cia ee 
a visttor must read. * 
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INTRODUCTION 


As more and more areas have been set aside for recreation or for their natural, 
scenic, or historic importance, the number of programs and facilities designed to 
interpret these areas has also grown. The prevalence and popularity of these interpre- 
tive services show that both administrators and visitors consider them worthwhile. Yet 
there are few criteria for determining how worthwhile. 


One of the great needs in the field of interpretation is to evaluate effectiveness. 
Do facilities and programs really have the effect they are intended to have?  Inter- 
preters have learned much by careful observation of visitors' reactions to these facilities 
and programs, but they could learn even more if visitors had a convenient way to register 
their reaction to interpretive services. Such "feedback" from visitors is needed to help 
the interpreter know how well he is doing and what changes might be useful. 


As one effort to provide this kind of feedback, a recording quizboard was developed 
and tested (fig. 1). This simply presents the visitor with four multiple-choice questions 
and records his answers with electric impulse counters. When the questions are based 
on information provided by interpretation, answers indicate what ideas are or are not 
reaching the visitors who play the quizboard. When changes are made in a presentation, 
the quizboard can quickly detect whether the communication of specific information was 


improved. 


Figure 1.--A, Recording quizboard in use at Ohanapecosh Visitor Center, Mount 
Rainter National Park; B, question arrangement on backlighted panel. 


CONSTRUCTION AND OPERATION 


The basic plan of the recording quizboard is simple. Four questions--each with 
four answer choices--are presented so that visitors can respond by pushing buttons. 
Unknown to the visitors, electric counters record their responses and provide a total 
count of correct and incorrect responses to each question. Thus, what visitors consider 
a game becomes a feedback and evaluation device for the interpretive planner. 


The circuitry of the quizboard is designed so that only one panel is lighted and only 
one set of buttons is energized at a time. When a question is correctly answered, that 
question panel darkens, the next question panel lights up, and a green panel reading 
"right answer" lights up (providing reinforcement to the visitor). 


Experience with an earlier version of the recording quizboard showed that some 
visitors--especially children--would push "correct" buttons repeatedly to see the green 
panel light up. Thus, the one-at-a-time panel lighting is important. This is accom- 
plished by a rotating relay. As shown in the wiring diagram (fig. 2), additional relays 
are used to handle the amperage needed for counters and to avoid arcing on the push- 
buttons. A time delay relay is used to assure proper timing of other components. 


Because the "correct" button for a question must be pushed before the next question 
panel will become operative, one ''correct"' answer counter is sufficient to record the 
number of four-question sequences completed; however, a separate "incorrect" counter 
is required for each question (fig. 3A). 


Correct and incorrect answer positions can be quickly changed by arranging 16 
plugs on a small wiring board (fig. 3A). A green plug for each panel corresponds to the 
"correct" answer position and three red plugs correspond to the "incorrect"’ answer 
positions for that panel. 


The quizboard was also designed for quick changes of question material. Questions 
can be typed (in oversize type) on tracing paper that is sandwiched between two sheets 
of acrylic plastic--a transparent gray sheet in front and a clear sheet in back (fig. 3B). 
The gray plastic sheet covering each question insures that it is readable only when 
illuminated from behind. The "sandwiches" are quickly slipped into the quizboard face 
which is removable (fig. 3C). 


Each question panel is lighted by two showcase lights mounted against reflectors 
made of sheet aluminum (fig. 3D). A 40-watt bulb illuminates the "correct answer" 
panel. Wood and sheet metal form boxes to separate panel lighting. 


The cost of components and materials for the recording quizboard was approxi- 
mately $215. 
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RELAY CONTACT — NORMALLY OPEN 


RELAY CONTACT — NORMALLY CLOSED. 
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WIRING DIAGRAM 


Figure 2.--Wiring diagram for the recording qutzboard. 
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TESTING ~ 


The quizboard was tested at the Ohanapecosh Visitor Center in Mount Rainier 
National Park during August and September of 1969 and late July and August of 1970. 
This is a small visitor center devoted to interpreting the interrelationships of various 
life forms within the lowland forest portion of the Park. Exhibits are of excellent quality; 
and except for those designed to orient the visitor, all are pertinent to the central theme 
of the visitor center. 


The 10 questions shown in table 1 were written to measure visitors' retention of 
specific information presented in the exhibits. The main purpose of testing was to see 
if the quizboard could distinguish between the effectiveness of different presentations. 
Questions were therefore based on a variety of exhibits incorporating different presenta- 
tions. Specific comparisons were made between audio and text presentations and between 
various combinations of three-dimensional (3-D) and such "flatwork"' presentations as 
photos and drawings. Questions 1 and 2 (for taped narration) were based on the recorded 
message of the key exhibit of the center, a diorama presenting mounted mammals and 
birds in a lifelike setting. When a button at this diorama was pushed, a recording ex- 
plained predator-prey relationships and interactions between the plant and animal com- 
munities of the forest. Lighting was coordinated with the message to illuminate which- 
ever animals were being discussed. ; 


Questions 3, 4, and 5 (for 3-D and text) were based on the texts of small signs 
around a relief model of Mount Rainier National Park. Questions 6, 7, and 8 (for com- 
binations of 3-D, flatwork, and text) were based on texts in three display cases inter- 
preting the three major trees of the area--Douglas-fir, western redcedar, and western 
hemlock. These cases included photographs and drawings along with such 3-D objects 
as upright sections of bark, mounted birds, and Indian artifacts. Questions 9 and 10 
(for flatwork and text) were based on panels that included only photographs, drawings 
or paintings, and text. 


For the 1969 tests, these questions were used in three sets of four as follows: 


Results for 1969 were distorted because of a faulty counter but suggested that 
visitor responses changed after Labor Day, when an older group predominated in the 
adjacent campground. After Labor Day the proportion of correct answers went up for 
all questions. The 1969 results also indicated that the position of the correct answer 
among the four choices might be affecting results. 


Because of the 1969 results, procedures were changed in 1970. To eliminate 
position effects, each of the 10 questions was written four ways, with the correct 
answer placed in each of the four possible positions. This gave a total of 40 questions 
presented in 12 sets of four. These were presented so that (1) two versions of the same 
question were never used at the same time, (2) a version of question 1 was included in 


*(pazlolpey! 8) BOIOYO joor1100) 


p pue ‘o ‘gq ‘ev suoytsod ul zaMsuL jo9I109 OY) YIM SUOTSA9A SDuIplAoad *sao!oyo ToMsuK Jo S]JUMUTADULAAL JUBILOJJIP ANOJ YIM posn svM suoljsanb QL ay) JO youo ‘OL6L UL 


p 


q 


D 


v 


Q Oo FD 


q 


<M 


3) 


Q 


Lov) 


uoljISsod 


sjuapou 


sjoasul 
surmey) pue Suizaoay 


udungy 


jSaroyj oanjeut 

v Jo apeys aq} ul sSuljpaes 
Alj-Sulsnog uvy} ta}Jaq AAtAaNS 
sSutjpeas yooyway asnvoaq 


Alj-sepsnog uvyy 
19}SvjJ SMOAS YOO] WAY asnvodac{ 
Uj-supsnog uvyy 
IIBUoO] SAAT] YOO] Way asnvoaq 


sjuase dsAtjoOnNaysap Tayo 
pue ‘sjoasul ‘arty 0} JURISISOI 
aIouUl ST yOo|Way asneoeaq 


auld a}1yM U1a}SAM 
Itj-se|snoq 
yoo|uay ulaysoM 


salty any 


Jasevue) urla}sam 
Avl Aeas 
IIYOJ poyeys-poa 


yeaqsois Suruadae 


Iepsopad U1a}SaM 
yoo|Wey ura]sem 
Alj-se[snoqg 


lap|e par 


sao10yo 1aMsuy 


L 


are SNUNY OF 19441] 
sutonpaa ut syuadeu jedioursrd 


Ij-sujsnog sovjdea 0} spuay 
yoo;way SuoIssadons jSsatoj Uy] 


JO ssul|paes 

1oj vould Surjaeys v aptaoid 
uayjo ‘oinjstow puke sjyuetaynu 
ITIY} YM ‘SdO] 3urjj{0Y 


ayy St [[Iq AAvay sit 
UJIM poss syoRIO jLY) Palq V 


jo yaeq 
snorqty oy} Woy Sapoyae 
Iayjo pue ‘Suryjojo ‘syayseq 
ape JSOMYION 94} JO SuRTpUT 


uoljseant 


Or 


6 


Iaquinn 


p 


uorjisod 


| 
| 


| 


"Sf ay) UL OUBOIOA 
A|ayjzou jSOW ay} SI 


"Ss" ayy ul purm 
papsoooa jsoys1y ay) sey 


‘S*f) 94) Ul Wass 


Jalovls jsodav] oy) sey 
‘sn oy) 

Ul ULBJUNOWW jSa]][B} AY) St 
adpo] oy) ye UaYyo yy 

A} 10} advaoys pjoo sv 
savaq Surjeuraqry Aq 


saaqui} |o 
poeuoorew 10j adnjaa e su 


Yotvasad dapoasd 104 


9TGL Ut poystiqeise 
906T Ut payst|qryse 
6681 Ul paysi{qeisa 


O68T Ul payst|qeisa 


uyAvU ayy 
yeaqoq 24} 
asnoid ay} 


9j0A00 9} 


payoajje aq jou pynom 


aonpoad pjnoo ysa10j ay 
uey} a10w Surjee Aq Ajddns 
pooj Alay} AOajsap pynom 


Snoxeurnu d10ul yOnUI aq plnoM 


Ay}eey pue yey aq pue 
yonw os A1ioM jou pjnom 


sao1oyo Jamsuy 


/t 


JOU BY 


juno *eyxSe|y JO apisyno S 


pasn usaq aAvy azaruley juno, 


JUNO 


jo }WWuUINs 94) WB SUTAAKD v 


SEM OILY 
*“ssaiduog jo joy ue Aq 


payst|qejse st y1eq [LUCIEN You] = g 


SI saliiaq Juijea punoj aq 
0} A[exI| JSOUL BANjeeID BY z 


jsa10j ay} JO sjewiue 


Surjeo jueld ay} ‘sjewiue 
Suijea ysaj Ou 210M 9104} JI I 


uoljsan zequinn 


7123499 JOUSIA YSOIadeueYyOC ‘pseogzinb Bulpsozas ay} YUM pasn suolsang --'| aqeL 


each set (causing each version to be used in three of the 12 sets), and (3) all other ques- 
tions were used just once. Within these constraints, questions were assigned to question 
sets at random. 


Each question set was used until the "correct" counter registered at least 500 
counts, insuring approximately even amounts of play for each set of questions. Data 
accumulated so rapidly that all question sets were used twice. 


RESULTS 


Proportions of correct answers for the 10 basic questions and four answer positions 
are shown in figures 4 and 5. These results correspond to interest patterns found in an 
interview study in which visitors at Ohanapecosh and four one visitor centers were 
asked to designate the exhibits they found most interesting.— The exhibits shown by 
interviews to be the most and least interesting were those the recording quizboard 
showed to have the highest and lowest retention of information. 


Statistical analyses were made to test for dependable differences (a) between 


specific groups of questions, (b) among all questions, and (c) among answer positions, 
These tests indicated that visitors remembered more from the exhibit with taped 
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Figure 4.--The effect of answer position for the 
recording quizboard. Data for all 10 questtons 
having the correct answer tn posttton 1 were 
combined, as were data for each set of 10 ques- 
ttons having correct answers in postttions 2, 3, 
and 4, respectively. Stnee equal amounts of 
data were used for all 40 questions, this group- 
tng averaged out the effects of content within 
each set of 10 questtons. 


1/ Randel F. Washburne. Visitor response to interpretive facilities at five visitor 
centers. Unpublished Master's thesis, University of Washington, Seattle. 91p., 1971. 


Question 


The creature most likely to be found eating 


If there were no flesh eating animals, the 

plant eating animals of the forest would re es ar 
destroy thetr food supply by eating 

more than the forest could produce. 


Each National Park is established by act of 
Congress. Mount Rainier was established dich devin oehuise hier ae I 


in 1899. 


Rotting logs, with their nutrients and moisture, 
often provide a starting place for seedlings gia tint Seiad Toe ey ale | 


of western hemlock. 


Indians of the Northwest made baskets, 
clothing, and other articles from the fibrous Be sincere, (hh onmobtwoagh Me aL 


bark of western redeedar. 


Outside of Alaska, Mount Rainier has the aie 
largest glacter system in the U.S. 

In forest succession, hemlock tends to 

seedlings survive better than Douglas-fir 

seedlings tn the shade of a mature forest. 

Principal agents in reducing litter to humus he aes, es 
are fungt. 

Caverns at the summit of Mount Rainier have nego 5 2) bc Ree 
been used as a refuge for marooned climbers. 


A bird that cracks seed with its heavy bill ee 
is the evening grosbeak. 
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Figure 5.--Results obtained by the recording quiab 1 J r 
Mount Ratnter Nattonal Park, 1970. Bars indicate correct an 
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in whieh one out of four plays was etther random or systematte and anoth 
out of four plays was correct because answers had been memorized during I 


playing of the quizboard. 
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narration (represented by questions 1 and 2) than from exhibits with written texts. 
Specifically, the tests permit us to be 99 percent sure that correct answer proportions 
for questions 1 and 2 combined are dependably higher than for all other questions com- 
bined. For comparisons among all questions, the tests permit us to be 95 percent sure 
that the correct answer proportion for question 2 is dependably higher than the correct 
answer proportion for question 4, 7, 9, or 10. 


Tests indicate a 90-percent likelihood that the position of the correct answer affects 
the proportion of answers that are correct--some visitors push answer buttons system- 
atically, as from top to bottom. 


A particularly interesting result from this study was the popularity of the quizboard 
with children. Although the exhibits at Ohanapecosh are of excellent quality, the quiz- 
board was the only item in the entire visitor center a child could manipulate. Within 
seconds after its installation, it became, for children, the most popular exhibit in the 
center. 


DISCUSSION 


Results show that the recording quizboard can tell the interpretive planner which 
questions are being answered correctly most often and can provide feedback that helps 
him determine the relative effectiveness of various exhibits or other interpretive efforts. 
Such feedback should be especially helpful for determining whether a change in a presen- 
tation makes it more effective or less so in conveying specific information to visitors. 


The quizboard, however, cannot always identify the exact reason a presentation 
is effective. Interpretive planners often seem to use their most effective techniques 
for their most exciting stories. For example, at the Ohanapecosh Visitor Center, 
questions based on the diorama with a recorded message were answered more correctly 
than others. Yet this could be explained by the use of a taped message, by exciting sub- 
ject matter (predators and prey), by exhibit size (largest in the building), by realism 
(unmatched by other exhibits in the building), or even by the possibility that visitors 
knew much of the subject matter before seeing the exhibit. 


Because visitors who play the recording quizboard are a self-selected group, 
results will not represent the average of all visitors. Nevertheless, they may still 
provide the best available indication of what visitors are learning. 


Differences between answers are reduced but not obscured by both repeated play 
and random or systematic play of the quizboard. At the Ohanapecosh Visitor Center, 
repeated play by children from the adjacent campground was especially common and 
accounted for a substantial portion of the total data. The usual pattern for a young 
"repeater" was to play until he learned the correct answers, play for several correct 
sequences, and then find someone else to watch him "'get them all right." 


To examine the effects of nonserious playing, I made computations for assumed 
amounts of such playing. For purposes of illustration, computations were made as 
if 25 percent of all answers were repeated plays in which all answers were correct 


because of prior memorization and as if another 25 percent of the answers resulted 
from random or systematic playing. 2 


The lines across the bars of figure 5 show results adjusted to exclude these 
assumed levels of nonserious play. Note that, in terms of correct answers, nonserious 
play does not change the order of questions but simply decreases differences between 
questions. This indicates that recording quizboards will give the most sensitive tests 
of interpretive effectiveness when located where they will be used primarily by transient 
visitors who are unlikely to play them repeatedly. In locations that encourage repeated 
play, as near overnight accommodations that hold visitors in an area for several days, 
results will be conservative and will show smaller differences than could be expected 
if all playing were Serious. 


Because playing the quizboard is obviously rewarding to many visitors, especially 
children, it can focus attention on specific information, thereby serving a teaching as 
well as a feedback function. The next step is to harness this reward pattern to enhance 
learning of more detailed concepts than are suited to quizboard question panels. A study 
of these possibilities is underway. 


CONCLUSIONS AND RECOMMENDATIONS 


1. Recording quizboards can be effective for obtaining feedback from visitors to 
indicate what they know or have learned. Quizboards should be especially good 
for detecting the effects of changes in interpretive facilities or presentations. 


2. For the questions it presents, a recording quizboard can detect real differences 
in proportions of correct responses made during serious playing, even if many 
visitors either play it repeatedly after memorizing answers or play it randomly 
or systematically. The effect of nonserious play is to reduce differences between 
proportions of correct answers. However, the order of results will not be changed. 


3. The positions of correct answers are likely to affect proportions of correct res- 
ponses. Several versions of each question should therefore be written so that 
correct answers appear in each position for approximately the same number of 
responses. For four-choice questions, this requires four versions of each question. 
These should be rotated every 500 or 1,000 responses according to a randomly 
determined schedule. 


zy) Question 2 had a correct-answer ratio of 676/1000. If a quarter of all answers were 

correct because of prior memorization, the correct-answer ratio would change to (676 - 250) 
- (1000 - 250). Since all four answer positions were used an equal number of times, the 
effects of systematic and random answering would be identical. If another quarter of the 
responses were either random or systematic, one out of four answer choices (250/4 = 62.5) 
would be correct. This would further change the correct-answer ratio to (676 - 250 - 62.5) 

(1000 - 250 - 250) = 0.727, anincrease. At the other end of the scale, question 7 with a 
correct-answer retio of 521/1000 would become (521 - 250 - 62.5) : (1000 - 250 - 250) = 0.417, 
a decrease. Subtracting other assumed amounts of nonserious answering also increases differ- 
ences between correct-answer ratios for different questions. 


10 , 


— 


4, When a recording quizboard is first used in a new Setting, statistical tests 
should be made to determine how large differences between answers must be 
to be statistically significant. Thereafter, such tests should not be necessary. 
Approximately 1,000 responses for each version of each question should 
provide data with adequate stability for comparisons to be made between 
questions. 


STATISTICAL ANALYSIS 


Although chi-square analysis was originally planned for the recording quizboard, 
observations were not suitably independent. This lack of independence resulted from 
repeated playing by some individuals and from the fact that most people answered all 
four questions. 


Because chi-square analysis was not appropriate, an analysis of variance was made, 
In this, the ratio of right answers to total answers was computed for each version of 
each question, giving 40 ratios (four versions each for 10 questions). Each ratio resulted 
from approximately 1,000 answer choices for the version and question involved. Although 
this does not seem to provide efficient use of each observation, large numbers of obser- 
vations are easily obtained. 


Analyses were made both with and without weighting for the number of observations 
on which each ratio was based. Results of these analyses were essentially identical, 
and only the unweighted analysis is reported (table 2). 


Table 2.-- Analysts of variance for quizboard data, Ohanapecosh 
Visttor Center, 1970 


Source Sum of squares Mean squares P 


Questions y) 0.06372 0.00708 SIL ESE al: 
Answer positions 3 -01414 - 00471 2.61 <i li) 
Error 27 - 04887 - 00181 

Total 39 


This analysis permits us to be 99 percent sure that means for the ratios of right 
answers to total answers differ among the questions. The analysis also permits us to 
be 90 percent sure that means of these ratios differ according to the position of the 
correct answer. 


Comparisons 


Designed comparisons were made to test for differences between questions based 
on three different patterns of exhibits: a diorama with recorded narration, 3-D exhibits 
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with text labels, and flatwork exhibits with text labels. Coefficients for the comparisons 
are Shown in figure 6. 


Of the nine designed comparisons, two were significant at 95 percent or higher 
probability: Comparison 1 showed that the two questions based on the diorama with re- 
corded narration had significantly more correct answers than the eight questions based 
on text; comparison 9 showed that, for information presented on text labels in exhibits 
combining both flatwork and 3-D elements, visitors gave the correct answer to a question 
on Indians significantly more often than to a question on the evening grosbeak. However, 
since the grosbeak was a minor element in a large display case, no generalization seems 
justified by the significant result for the comparison. 


In addition to the designed comparisons, al] comparisons among means for the 10 
answer ratios were made using the Tukey test.— In this, the minimum significant 
difference D was computed as 


Q verror mean square/number of observations per mean = 4.88 v0. 00181/4 = 0.1039 


where Q is a table value for 10 treatments and 27 degrees of freedom. 


Only four differences between means exceeded this value. These were as follows: 


Question 7 Question 4 Question 10 Question 9 
Question 2 0. 1552 On t215 0. 1086 0. 1067 


Comparison Coefficients 
Audio | 3-D only and text | 3-D plus flatwork and text Flatwork and text 
+ + === 
Question 1 2 | 3 4 5 6 7 8 9 10 
-- 4 
Mean -62195 .67628 LL -60960 .55475 .57867 -58540 .52105 .59730 -56958 .56770 
] +4 +4 -] -] -] -] -] -1 -1 -1 
2 +] } -1 ; 
3 | +1 +] +] +1 +1 +1 -3 -3 
4 | +] +] +] -] -] -1 
5 | +] -] 
| 

6 | | | +] +1 -2 
7 | | | +] =i] 
8 | +] +1 -2 
9 cap tlhe nat 


| | Ee 


Computations for comparisons 1 and 9, using procedures from p. 329-332 of George W. Snedecor (Statistical 


Methods, 5th ed., Iowa State College Press, Ames. 1956). (Note that means are multiplied by 4 to convert them 
to sums.) 
¥ > 27NR72 
Comparison 1: Sum of squares = L4(2-4878)+4(2.70512)-2.4384-2.2190-2.31468-2.3416-2.0842-2.3892-2.27832-2.2708] 
: 4(40) 
= 0.03707 p= oar = 20+ p<0.01 
2 
Comparison 9: Sum of squares = (2.3465 ose)" 
= 0.00828 F = oon = 4.5+ P <0.05 


Figure 6.--Destgned comparisons and orthogonal coeffictent sets for 
recording quizboard data. 


3/ George W. Snedecor. Statistical methods, p. 251-253. Ames, Iowa State Coll. 
Press, 1956. 
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s The mission of the PACIFIC NORTHWEST FOREST 
ae AND RANGE EXPERIMENT STATION is to provide the 
ae knowledge, technology, and alternatives for present and 
future protection, management, and use of forest, range, and 
related environments. 


Az; Within this overall mission, the Station conducts and 
oe stimulates research to facilitate and to accelerate progress 
ire toward the following goals: 
oe 1. Providing safe and efficient technology for inventory, 
a protection, and use of resources. 

ae 2. Development and evaluation of alternative methods 
wa ; and levels of resource management. 

; 3. Achievement of optimum sustained resource produc- 
—G tivity consistent with maintaining a high quality forest 

M environment. 

is + The area of research encompasses Oregon, Washington, 
> Alaska, and, in some cases, California, Hawaii, the Western 
we v States, and the Nation. Results of the research will be made 

ia available promptly. Project headquarters are at: 

ty Fairbanks, Alaska Portland, Oregon 

a Juneau, Alaska Olympia, Washington 

ay Bend, Oregon Seattle, Washington 

re Corvallis, Oregon Wenatchee, Washington 

eS La Grande, Oregon 
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is dedicated to the. 


2 management of the 
Nation's forest re Helds of wood, water, 
forage, wildlife, ane ste . gh forestry research, 
cooperation with the te " fe forest owners, and 
management of the Nai abe: : 

strives — as directes 
greater service to a gr 


